Augmented reality of a building

ABSTRACT

The present disclosure relates to an augmented reality system including one or more of a building, sensor(s), a gateway, a server, and a monitoring computer. The augmented reality system may further includes drones that capture video or other images of the building. The monitoring computer gathers information from the sensor(s) and/or drones and facilitates the creation of an augmented view of the building to provide intelligence to emergency responders.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisional Pat.Application No. 16/019,366, filed on Jun. 26, 2018, which claims thebenefit of and priority to U.S. Provisional Pat. Application No.62/525,496, filed on Jun. 27, 2017, each of which is incorporated byreference in its respective entirety herein.

BACKGROUND

The following description is provided to assist the understanding of thereader. None of the information provided or references cited is admittedto be prior art. Most building structures are prone to catching fire.These building structures may be equipped with one or more smokesensors. In the event of a fire, the smoke sensors are configured todetect smoke and sound an audible alarm. Even with the alarm, occupantsmay be trapped within the building structure and may need to be rescued.Not knowing the layout of the building structure and where the occupantsare trapped within the building, the rescue of the occupants may take along time. The layout of the building may even change as the firespreads within the building, further complicating rescue efforts. Insome cases, the occupants are not able to be rescued primarily due tothe treacherous, hazardous, and rapidly changing conditions within thebuilding structure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an augmented reality system of a building,in accordance with at least some embodiments of the present disclosure.

FIG. 2 is a flowchart of a method for monitoring conditions of abuilding, in accordance with at least some embodiments of the presentdisclosure.

FIG. 3 is a flowchart of a method for generating an augmented reality ofa building, in accordance with at least some embodiments of the presentdisclosure.

FIG. 4 is a block diagram of a computer system, in accordance with atleast some embodiments of the present disclosure.

The foregoing and other features of the present disclosure will becomeapparent from the following description and appended claims, taken inconjunction with the accompanying drawings. Understanding that thesedrawings depict only several embodiments in accordance with thedisclosure and are, therefore, not to be considered limiting of itsscope, the disclosure will be described with additional specificity anddetail through use of the accompanying drawings.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented here. It will be readily understood that the aspects of thepresent disclosure, as generally described herein, and illustrated inthe figures, can be arranged, substituted, combined, and designed in awide variety of different configurations, all of which are explicitlycontemplated and make part of this disclosure.

Buildings such as homes, schools, theatres, office buildings,warehouses, etc. often include devices that monitor one or moreconditions of the building. For example, smoke sensors can detect alevel of smoke in a room, occupancy sensors can detect the number ofpeople in a space, temperature sensors can detect the temperature of aroom, etc. Such sensors can be configured to monitor for undesirableconditions such as a fire, high levels of toxic or explosive gasses(e.g., carbon monoxide, methane, etc.), floods, high levels of humidity,etc. The sensors can transmit information regarding their sensedconditions to a central processing system that can monitor theinformation and detect the undesirable conditions.

Once the undesirable conditions are detected by the central processingsystem, appropriate personnel can be contacted and notified. Forexample, a homeowner or a police station can be contacted when thesystem detects an intruder. In another example, the fire department canbe notified if a fire is detected. However, real-time data from thesensors may not be available to emergency personnel during theundesirable condition. For example, once a fire truck arrives at abuilding with a fire, the firemen may know that there is a fire, butthey may not know details about the condition such as which rooms are onfire, where people are located within the building, what the floorplanof the building is, etc. Various embodiments described herein facilitatetransmission and effective communication of the data detected by thesensors to the emergency personnel.

In an illustrative embodiment, once emergency personnel arrive at abuilding with various sensors, the emergency personnel can deploy dronesto gather the information from the sensors. For example, the sensors maywirelessly transmit information such as temperature, occupancy, smoke,etc. The drones can fly or otherwise maneuver into positions that areclose enough to the sensors to detect the wireless transmissions. Thedrones can transmit the information to a computing system available tothe emergency personnel. The computing system can compile the variousinformation into a virtual environment that provides an augmentedreality to the emergency personnel. The virtual environment can helpfacilitate the emergency personnel to most effectively handle theundesirable conditions.

Referring now to FIG. 1 , an illustrative block diagram of an augmentedreality system 100 is shown, in accordance with at least someembodiments of the present disclosure. The augmented reality system 100,as illustrated, includes a building 105, sensors 110, a gateway 115, aserver 120, drones 125, and a monitoring computer 130. In alternativeembodiments, additional, fewer, and/or different elements may be used.

The augmented reality system 100 is implemented in the building 105.Within the building 105, the augmented reality system 100 includes thesensors 110 such as sensors 135, 140, 145, 150, 155, and 160. In anillustrative embodiment, the sensors 110 are located throughout thebuilding. In alternative embodiments, the sensor 110 can be located instrategic locations based on the conditions monitored. For example, awater level sensor can be located in a basement to monitor for floodingof the basement. In the embodiment illustrated in FIG. 1 , six of thesensors 110 are shown. However, in alternative embodiments, any suitablenumber of sensors may be used such as greater than six or fewer thansix.

Each of the sensors 135, 140, 145, 150, 155, and 160 are configured tocommunicate with the gateway 115 via communication links 165 to transferdata collected by the sensors to the gateway. The sensors 135, 140, 145,150, 155, and 160 may communicate any suitable information to thegateway. For example, each of the sensors 135, 140, 145, 150, 155, and160 can communicate an identification (ID) of the respective sensor andan indication of the condition sensed by the respective sensor (e.g., atemperature, a humidity level, a level of smoke obscuration, etc.).

In an illustrative embodiment, the gateway 115 communicates with theserver 120, via communication link 170, to transmit the sensor datareceived from the sensors 135, 140, 145, 150, 155, and 160 for furtherprocessing and cataloguing. In the embodiment shown in FIG. 1 , theserver 120 is remote from the building 105. For example, the server 120may be implemented in a cloud-based computing system. In alternativeembodiments, the server 120 may be located within the building 105. Forexample, the gateway 115 and the server 120 may be implemented in asingle device that communicates with the sensors 135, 140, 145, 150,155, and 160 to process the information from the sensors.

The augmented reality system 100 also includes the drones 125 that areused to capture videos and/or images of the building 105 during anemergency. In an illustrative embodiment, the videos or images are usedto help devise an emergency response plan, as discussed further below.The drones 125 may also be used to capture information transmitted fromthe sensors 135, 140, 145, 150, 155, and 160. The drones 125 maytransmit the sensor information and the images to the monitoringcomputer 130. The monitoring computer 130 can gather the information andpresent the information to users (e.g., emergency personnel) in a mannerthat provides the users with an augmented reality of the building 105.

It is to be understood that only some components of the augmentedreality system 100 are shown herein and that several other components,such as, routing mechanisms, wiring, processing units, installationunits, etc. may be used and are within the scope of the presentdisclosure. Likewise, although not shown, the building 105 may havewalls, a roof, windows, doors, a foundation, and other features found inbuildings. Furthermore, the building 105 may be any of a variety ofstructures such as residential houses, commercial buildings, warehouses,hospitals, stores, factories, hotels, arenas, stadiums, and any othersuitable building. While the present disclosure has been described forthe building 105, in other embodiments, the present disclosure may beimplemented in any other structure that is substantially permanent ortemporary, such as tunnels, caves, tents, stages, or any structure thatis monitored by the sensors 110.

With respect to the sensors 135, 140, 145, 150, 155, and 160, in someembodiments, the sensor 135 is a smoke sensor that is used to detectsmoke in the building 105, the sensor 140 is a motion sensor that isused to detect the presence of people or pets in the building in case offire, the sensor 145 is a door sensor to determine the status of doors(e.g., open, closed, locked, unlocked, etc.) throughout the building,and the sensor 150 is a window sensor to determine the status of windowsthroughout the building. Likewise, in some embodiments, the sensor 155is a temperature sensor to detect the temperature of a room and thesensor 160 is a wind sensor to determine the direction of wind in and/oraround the building. Notwithstanding that the sensors 135, 140, 145,150, 155, and 160 have been described as being a specific type ofsensor, in other embodiments, other or additional types of sensors, suchas, stair sensors, carbon monoxide sensors, location sensors, proximitysensors, pressure sensors, water sensors, humidity sensors, smokeobscuration sensors, and any other type of sensor that may be needed ordesirable to have to get information in case of a fire in the building105, may be used.

Further, although only one instance of each of the sensors 135, 140,145, 150, 155, and 160 is shown in the augmented reality system 100 ofFIG. 1 , in at least some embodiments, multiple instances of one or moreof those sensors 110 may be used in the building 105. Specifically, thenumber and location of the sensors 135, 140, 145, 150, 155, and 160within each room or area of the building 105 may depend upon the type ofsensor, the type of information that the sensor is configured to collectand convey, and/or the amount of information that may be needed ordesirable to have in devising an emergency plan. For example, if thesensor 145, which is a door sensor, is configured to detect the statusof a single door, one instance of the sensor 145 may be installed onevery (or selected) door(s) of a room or area in the building 105. Inanother example, if the sensor 145 is designed to determine the statusof every door in a room or area of the building 105 (e.g., by detectingdoors within a pre-defined range of the sensor), one only instance ofthat sensor may be needed in each room or area. Furthermore, if the doorin a room or area of the building 105 is a closet door, then thelocation of that door may not be pertinent to devising fire emergencyplan and, therefore, the sensor 145 may not be needed on the closetdoor.

Likewise, window sensors (e.g., the sensor 150) may be installed onevery (or selected) window(s) of the building 105 or only those windowsthat are on the outside walls of the building. Similarly, one instanceof a smoke sensor (e.g., the sensor 135) may be sufficient for everyroom, in some embodiments. However, for bigger rooms, it may bedesirable to have multiple smoke sensors. In some embodiments, multipleinstances of a sensor may be installed to merely provide redundancy,such that, in event that one sensor malfunctions, another sensor cankeep sensing. Thus, the sensors 135, 140, 145, 150, 155, and 160 may bestrategically placed throughout the building 105 to get information frominside the building to effectively and efficiently sense or detect anyfires in the building and rescue people, pets, and/or property frominside the building.

Furthermore, each of the sensors 135, 140, 145, 150, 155, and 160 may bedesigned to withstand temperatures found inside buildings (e.g., thebuilding 105) during fires without being damaged for a specified periodof time. For example, in some embodiments, the sensors 135, 140, 145,150, 155, and 160 may be designed to withstand at least 500° C. for atleast 30 minutes during fire in the building 105. In other embodiments,the sensors 135, 140, 145, 150, 155, and 160 may be designed withdifferent specifications for temperature and time. Additionally, each ofthe sensors 135, 140, 145, 150, 155, and 160 may have an identity codethat uniquely identifies the sensor and distinguishes that sensor fromother types of sensors. The identity code may be transmitted along withthe other sensor data. The identity codes may also be made available ina catalog form for operators reviewing the sensor data.

In some embodiments, the identity code may be dependent upon the type ofsensor. For example, in some embodiments, all smoke sensor type ofsensors may have one kind of identity code, all door sensors may haveanother type of identity codes, all temperature sensors may have yetanother type of identity codes, and so on. Within each type of identitycode, additional information may be provided to distinguish multipleinstances of the same type of sensor. Thus, for example, two smokesensors in the building 105 may each have an identity code with twoparts - a first part identifying the sensor as a smoke sensor and asecond part that distinguishes the first smoke sensor from the secondsensor. By virtue of assigning specific codes to specific types ofsensors, the operators(s) reviewing the sensor data are able todetermine from the identity code the type of sensor and how to interpretthe data being transmitted from that sensor. The operators(s) may alsobe able to sort and arrange data based upon the type of sensor to mosteffectively and efficiently devise the emergency rescue plan for anyoccupants of the building 105.

Furthermore, in some embodiments, the identity code may also includeinformation identifying the location of each of the sensors 135, 140,145, 150, 155, and 160. In other embodiments, the location informationof the sensors 135, 140, 145, 150, 155, and 160 may be providedseparately from the identity codes. In some embodiments, the locationmay be global positioning system (GPS) coordinates. In otherembodiments, if the floorplan of the building 105 is known, the locationmay be a description of the location of the sensors 135, 140, 145, 150,155, and 160. For example, a location description of a door sensor(e.g., the sensor 145) may be “front entrance on the main floor” or“entrance into the basement,” etc. By virtue of knowing the location ofeach sensor, the operator(s) may be able to review the data from varioussensors to determine fire patterns, how fire is spreading, and whichareas to focus rescue efforts on. The location of the sensors 135, 140,145, 150, 155, and 160 may also be catalogued, either as part of theidentity code or separately, whichever the case may be, and madeavailable to the operator(s).

In an illustrative embodiment, the sensors 135, 140, 145, 150, 155, and160 transmit sensor data to the gateway 115. In some embodiments, oneinstance of the gateway 115 can be provided in each room or area of thebuilding 105, while in other embodiments, one instance of the gatewaymay be sufficient for the entire building 105. In yet other embodiments,multiple instances of the gateway 115 may be desirable in each room orarea of the building 105. The number of the gateways (e.g., the gateway115), as well as the positioning of the gateways within the building 105may vary based on the operating range of the gateway and its ability tocollect sensor data from the sensors 135, 140, 145, 150, 155, and 160.In an illustrative embodiment, similar to the description above of thesensors 135, 140, 145, 150, 155, and 160, the gateway 115 may bedesigned to withstand high temperatures for a specified period of timeto enable the gateway to keep collecting and transmitting sensor datafrom the sensors 135, 140, 145, 150, 155, and 160 in case of fire.

Furthermore, in some embodiments, the gateway 115 may be configured totransmit the sensor data from the sensors 135, 140, 145, 150, 155, and160 instantaneously (or substantially instantaneously) as the sensordata is received from the sensors. In other embodiments, the gateway maybe configured to transmit the sensor data periodically. Likewise, thesensors 135, 140, 145, 150, 155, and 160 may be configured to eithersense their assigned condition constantly or periodically, as programmedwithin the sensors. In some embodiments, the gateway 115 is configuredto direct the sensors 135, 140, 145, 150, 155, and 160 to collect sensordata and transmit that data to the gateway.

In an illustrative embodiment, the sensors 135, 140, 145, 150, 155, and160 may transmit information to the gateway 115 while in communicationwith the gateway 115. If communication is interrupted, the sensors 135,140, 145, 150, 155, and 160 can buffer sensed information (e.g., alongwith a timecode) and can transmit the buffered information to thegateway 115 when communication is reestablished. Similarly, in anillustrative embodiment, the gateway 115 can transmit received sensorinformation to the server 120 while communication with the server. Ifthe communication path is interrupted, the gateway 115 can buffer thesensor information. The gateway 115 can transmit the bufferedinformation to the server 120 once communication with the server isreestablished.

The sensors 135, 140, 145, 150, 155, and 160 and the gateway 115 cancommunicate with each other via the communication links 165. Thecommunication links 165 may be any of a variety of communicationchannels or interfaces. The communication links 165 may be of any typethat is suitable for communicating with other sensors and/or the gateway115. For example, one or more of the communication links 165 may bewired communication links (e.g., jacks, antennas, transmitters,receivers, transceivers, wire terminals, electrical cables andconnectors, etc.) that communicate via wired communication protocols,such as TCP/IP, BACnet IP, BACnet MSTP, CAN, Modbus, USB, Firewire,UART, SPI, RS-485, a public switched telephone network (PSTN), or otherwired protocols.

Likewise, one or more of the communication links 165 may instead bewireless and communicate via one or more wireless communicationprotocols, such as Wi-Fi (including TCP/IP), Wi-Max, Bluetooth, LoRa,NFC, Zigbee, and the like. In some embodiments, one or more of thecommunication links 165 may include cellular or mobile phonecommunications, wireless radio channels, local area network (LAN),metropolitan area network (MAN), wide area network (WAN), world wide web(WWW) or the Internet, and the like. A combination of one or morecommunication interfaces discussed above may be used for thecommunication links 165 as well, in some embodiments. The communicationlinks 165 may also be a distributed intelligent network, in someembodiments. Further and similar to the sensors 135, 140, 145, 150, 155,and 160 and the gateway 115, the communication links 165 are designed tooperate in high temperatures found in case of a fire within the building105 for a specified period of time.

In addition to the sensors 135, 140, 145, 150, 155, and 160communicating with the gateway 115, in some embodiments, the sensors canalso communicate with each other. For example, if the sensors senseperiodically and if one smoke sensor detects smoke in a room, that smokesensor may alert other smoke sensors in the building to increasefrequency of sensing. Likewise, if the communication link 165 between aparticular one of the sensors 135, 140, 145, 150, 155, and 160 and thegateway 115 malfunctions, the sensor can transmit its sensed data toanother sensor, which may then transmit the other’s sensor’s data to thegateway. Thus, by enabling the sensors 135, 140, 145, 150, 155, and 160to communicate with one another, the augmented reality system 100provides redundancy in the system to account for malfunctions in thesystem. Such a network may be a self-healing network.

The gateway 115, in some embodiments, is also configured to include anidentity code and a location code. The identity and location codes ofthe gateway 115 may be useful when multiple instances of the gateway 115are installed within the building 105. In some embodiments, the locationcode of the gateway 115 may be part of the identity code. In someembodiments, the gateway 115 may transmit the identity code (and thelocation code if not part of the identity code) to the server 120 alongwith each transmission of the sensor data. Alternatively, oradditionally, the identity code and the location code may be cataloguedand made available to the operator(s).

In some embodiments, the gateway 115 may be programmed to automaticallycommunicate with an emergency response center (not shown) if the gatewayreceives certain types of information from the sensors 135, 140, 145,150, 155, and 160. For example, if the gateway 115 receives sensor datathat indicates that the temperature in a room or area of the building105 is significantly high and the data from smoke sensor sensors detectssmoke in the same room or building, the gateway 115 may conclude thatthe room or area of the building is on fire. The gateway 115 may thenautomatically communicate with an emergency response center. Conditionsthat may cause the gateway 115 to contact the emergency response centermay be programmed within the gateway. The emergency response center mayinclude a 911 call center, a fire department, police department, otheremergency response service, or a combination thereof. In an alternativeembodiment, the gateway 115 may not include such programming, and theserver 120 may analyze the information and automatically contactemergency personnel.

In an illustrative embodiment, the gateway 115 transmits the sensor datafrom the sensors 135, 140, 145, 150, 155, and 160 to the server 120 viathe communication link 170. The communication link 170 can be anysuitable communication link, such as any of the communication protocolsmentioned above with respect to the communication links 165. In anillustrative embodiment, the communication link 170 is a wirelesscellular link. In some embodiments, the sensors 135, 140, 145, 150, 155,and 160 may also be configured to bypass the gateway 115 and communicatedirectly with the server 120. Such a functionality may be used, forexample, when the gateway 115 malfunctions or one or more of thecommunication links 165 are ineffective. In such an example, the sensors135, 140, 145, 150, 155, and 160 can transmit the sensor data directlyto the server 120 via communication links that are similar to thecommunication link 170 (e.g., via cellular communications). The server120 may be located in a remote location (such as the emergency responsecenter or a cloud computing network) or another location accessible bymultiple emergency response centers. Data transmitted to the server 120may be stored in a cloud storage. In other embodiments, other oradditional mechanisms to store, manage, and back-up data may be used.Further, although not shown, multiple augmented reality systems (e.g.,the augmented reality system 100) connected to various other buildingsand structures may be connected to the server and store data in thecloud storage.

In an illustrative embodiment, the server 120 includes a variety ofcomputing elements. For example, the server 120 may include memoryhaving one or more devices (e.g., memory units, memory devices, storagedevices, etc.) for storing data and/or computer code for completingand/or facilitating various processes described in the presentdisclosure. Such memory may include volatile and/or non- volatile memoryincluding, random access memory (RAM), read-only memory (ROM), dynamicrandom access memory (DRAM), programmable read only memory (PROM),erasable programmable read only memory (EPROM), electrically erasableprogrammable read only memory (EEPROM), flash memory, hard drivestorage, optical memory, or any other suitable memory for storing data,software objects and/or computer instructions. Memory may also includedatabase components, object code components, script components, or anyother type of information structure for supporting the variousactivities and information structures described in the presentdisclosure. Other types of storage media, for example, compact disc(CD), digital video disc (DVD), floppy discs, Blu-ray discs, magneticstorage, computer readable media, or other electronic storage media, maybe used within or in conjunction with the memory of the server 120.

In an illustrative embodiment, the server 120 includes a processing unitcapable of executing computer code for running one or more processesdescribed herein. In some embodiments, the processing unit may include adigital signal processor (DSP), such as, a general-purpose stand aloneor embedded processor, or a specialized processing unit. In general,processing unit of the server 120 may be of any type that is suitablefor a server system. Additionally, in at least some embodiments,multiple instances of the processing unit may be connected together atleast indirectly and utilized in combination with one another to performvarious functions described herein. Multiple server units incommunication with one another may also be used in other embodiments.

The processing unit(s) of the server 120 may be configured to process avariety of program instructions and data, in accordance with theembodiments of the present disclosure. These program instructions anddata need not always be digital or composed in any high-levelprogramming language. Rather, the program instructions may be any set ofsignal-producing or signal-altering circuitry or media that may becapable of performing functions, described in the present disclosure.Additionally, the server 120 may include other hardware, software, andfirmware components that are commonly present in devices configured asservers and communicating with devices, such as the gateway 115. Withrespect to the communication link 170, that communication link may besimilar to the communication links 165 to the extent that thoseinterfaces are suitable for communication between the gateway 115 andthe server 120. Other types of communication channels may be used aswell.

As shown in FIG. 1 , the augmented reality system 100 includes thedrones 125. In an illustrative embodiment, the drones 125 can beunmanned aerial vehicles (UAV), which are small aircrafts configured tobe programmed and remotely operated by an operator to carry out one ormore functions. In an alternative embodiment, any other suitable meansfor collecting images and sensor data may be used, such as terrestrialdrones, fixed location devices (e.g., that communicate with the sensors110), handheld devices, etc. In an illustrative embodiment, the drones125 can be used to capture live videos and images of the building 105 inthe event of a fire in the building and transmit those videos and imagesto a drone operator, the emergency response center, the server 120, or acombination thereof. The images can be used, for example, to devise anemergency rescue plan. To capture the videos and images of the building105, each of the drones 125 has installed thereon a camera 175. Thecamera 175 may be configured to be controlled by the drones 125 and/orby the operator(s) controlling the drones.

The camera 175 can be configured to capture images. In some embodiments,the camera 175 can capture two-dimensional images. In other embodiments,the camera 175 can capture three-dimensional images. In an illustrativeembodiment, the camera 175 can capture videos. In an illustrativeembodiment, the camera 175 can be a still-image camera, a video camera,or any type of camera suitable for capturing high resolution aerialimagery. Furthermore, the camera 175 can be configured to capture colorimages, black-and-white images, filtered images (e.g., a sepia filter, acolor filter, a blurring filter, etc.), images captured through one ormore lenses (e.g., a magnification lens, a wide angle lens, etc.), etc.In some embodiments, camera 175 (or the drones 125 or the operator(s) ofthe drones) can modify one or more image settings or features, such ascolor, contrast, brightness, white scale, saturation, sharpness, etc. Inan illustrative embodiment, the camera 175 captures infrared images orhigh resolution images in low light. For example, the camera 175 can bea thermal imaging camera.

The drones 125 can communicate via a communication network (not shown)that facilitates the videos/images captured by the camera 175 of thedrones being transmitted to the monitoring computer 130 in real time.For example, the drones 125 can communicate via wireless communicationswith the monitoring computer 130.

Although not shown in FIG. 1 , the drones 125 can have all thecomponents that are commonly found in drones. For example, the drones125 may have one or more motors, rotors, wings, sensors, computers,power supplies, actuators, software, etc. All such components and otherelements commonly found in drones are contemplated and considered withinthe scope of the present disclosure.

In the event of an undesirable condition such as a fire, the emergencyresponse crew (e.g., fire fighters) may release the drones 125 aroundthe building 105. In some embodiments, one instance of the drones 125 isreleased for each side of the building. For example, five of the drones125 are released - one each for the four sides of the building 105 andone for the roof. In other embodiments, multiple drones (e.g., thedrones 125) for each side or roof of the building 105 may be used.Alternatively, fewer than five drones may be used around the building105. In an illustrative embodiment, the drones 125 are arranged aroundthe building 105 after being released and capture videos and images ofthe building and transmits those videos and images to the emergencyresponse crew.

The emergency response crew is able to view the sensor data from thesensors 135, 140, 145, 150, 155, and 160 via the monitoring computer130. The emergency response crew can receive the data from the sensors135, 140, 145, 150, 155, and 160 either directly through the sensors,through the gateway 115, and/or through the server 120. In someembodiments, the drones 125 receive the sensor data from the sensors135, 140, 145, 150, 155, and 160 (e.g., either directly from the sensors110, the gateway 115, and/or the server 120). Upon receiving thevideos/images and the sensor data from the drones 125, the sensor datais overlaid on the videos/images to obtain an augmented view of thebuilding 105.

For example, during a fire, the augmented view may provide intelligenceto the emergency response crew regarding the layout of the building 105,whether any people or pets are inside the building, where the fire isconcentrated, how the fire is spreading, which entryways would be bestfor the emergency crew to use for entering the building, etc. Theemergency response crew can use this information to devise an actionplan intelligently and to effectuate the plan most effectively andquickly. After the fire, the augmented view may provide data to help theemergency response crew determine the cause of fire or events thatoccurred during the fire. For example, the emergency response crew (orother personnel) may review the sensor data from the sensors 135, 140,145, 150, 155, and 160 to identify a room or area of the building 105that first reported higher temperatures associated with fires and/orsmoke. The emergency response crew (or other personnel) may use datafrom the other sensors or investigate in the identified room or area ofthe building 105 to determine the exact cause of fire.

In an illustrative embodiment, the sensors 135, 140, 145, 150, 155, and160 transmit unencrypted data using an omnidirectional antenna. Forexample, the sensors 135, 140, 145, 150, 155, and 160 can transmit viaWi-Fi or Bluetooth ®. The drones 125 can each be able to receive thecommunications from the sensors 135, 140, 145, 150, 155, and 160. Forexample, each of the sensors 135, 140, 145, 150, 155, and 160 canbroadcast a signal at regular intervals. Each signal can includeidentification information of the respective sensor 135, 140, 145, 150,155, and 160 and an indication of the sensed condition (e.g., atemperature, a humidity level, an amount of carbon monoxide, etc.). Thedrones 125 can receive the broadcasted information and transmit theinformation to the monitoring computer 130. For example, the drones 125can be arranged around the building 105 such that at least one of thedrones 125 is in communication range with each of the sensors 110 in thebuilding 105. In alternative embodiments, the drones 125 may be arrangedaround the building 105 to be in communication range with as many of thesensors 110 as practical. For example, when one or more of the sensors110 are located in the center of the basement, those sensor(s) may nothave enough transmitting power to transmit a signal to the drones 125.

In an illustrative embodiment, the monitoring computer 130 receives thesensor data and images from the drones 125 and generates an augmentedview of the building. For example, the monitoring computer 130 cangenerate a three-dimensional environment of the building using images(e.g., real-time images) from the drones 125. Overlaid or otherwiseembedded into the environment can be sensor data. For example, themonitoring computer 130 can receive a floorplan of the building (e.g.,from the server 120). The sensor data can be displayed on the floorplancorresponding to the room or area that the data corresponds to. Forexample, a temperature sensor in a master bedroom can be displayed onthe floorplan or a three-dimensional representation of the floorplan atthe location of the master bedroom.

FIG. 2 is a flowchart of a method of monitoring conditions of abuilding, in accordance with at least some embodiments of the presentdisclosure. As discussed above, the augmented reality system 100includes one or more drones (e.g., the drones 125) that an emergencyresponse crew (e.g., fire fighters) can release around the building 105when the building catches fire. The drones 125 can capture live streamvideos and images of the building (e.g., the building 105) and send thevideos and images to the monitoring computer 130. The drones 125 cangather sensor data from one or more of the sensors 135, 140, 145, 150,155, and 160 and transmit the sensor data to the monitoring computer130.

In a step 205, the method 200 is initiated. For example, in response toan undesirable condition (e.g., a building fire, a burglary, etc.),drones such as the drones 125 can be released. In an illustrativeembodiment, an occupant or observer of the building may report theundesirable condition to an emergency response center (e.g., firestation or 911 operator). Alternatively or additionally, the gateway 115and/or the server 120 may determine that the undesirable conditionexists within the building 105 based on sensor data received from thesensors 135, 140, 145, 150, 155, and 160 and automatically notify theemergency response center. Upon arrival of the emergency response crewfrom the emergency response center at the building 105, the crewreleases the drones 125 around the building. In an illustrativeembodiment, before releasing the drones 125 around the building 105, thedrones are programmed to capture videos and images of the building 105from multiple directions and angles and to receive informationtransmitted by the sensors 135, 140, 145, 150, 155, and 160. Forexample, the emergency response crew can program the drones 125 eitherat the site of the building 105 or while en route to the building 105.

In an illustrative embodiment, once at the building 105, the emergencyresponse crew releases the programmed drones 125. The number of thedrones 125 that are released may vary from embodiment to embodimentdepending upon the size, height, and other features of the building 105,as well as the size or location of the undesirable conditions.

At a step 210, a destination of each of the drones 125 is located. Forexample, each of the drones 125 can receive from the monitoring computer130 a location to travel to. In such an embodiment, the monitoringcomputer 130 can determine the location that the drones 125 shouldtravel to from information available from the Internet (e.g., onlinemaps or databases) or from the server 120. For example, the monitoringcomputer 130 can determine that the drones 125 should be spread aroundthe building 105 in a manner that puts the drones 125 withincommunication distance to as many sensors 110 as possible whilemaintaining a safe distance (e.g., far enough away from a fire).

After being released, the drones 125 travel to their assigned areas ofthe building 105 at a step 215. Thus, based upon their assignment, thedrones 125 travel to the roof, sides, or the corners of the building105. In some embodiments, the drones 125 may be programmed to just hoverover their assigned areas, while in other embodiments, the drones may beprogrammed to continuously fly around the building (or their assignedarea of the building) to capture videos and images. In addition totraveling to their assigned area of the building to capture videosand/or images, at step 220, the drones 125 may receive sensor data fromthe sensors 135, 140, 145, 150, 155, and 160 that are within thebuilding 105.

In some embodiments, the drones 125 have the capability to directlyconnect to one or more of the sensors 135, 140, 145, 150, 155, and 160,the gateway 115, and the server 120 to get the sensor data. In anillustrative embodiment, the drones 125 may be programmed to overlay thesensor data over the captured videos and/or images to create theaugmented view and transmit the augmented view to the monitoringcomputer 130. For example, in an illustrative embodiment, the drones 125can determine a location of the sensors 135, 140, 145, 150, 155, and 160based on a signal received from those sensors. In an illustrativeembodiment, a signal strength of the sensors 135, 140, 145, 150, 155,and 160 from each of two or more of the drones 125 can be used todetermine a location of the sensor 110 (e.g., biangulation,triangulation, etc.) In an illustrative embodiment, the drones 125 cancontinuously update the augmented view with the updated sensor data andthe live recorded videos and/or images. Thus, the drones 125 can providea real-time (or substantially real- time) augmented view of the building105 and of the fire engulfing the building.

The drones 125 record videos and/or images of the building 105 at step225. In an illustrative embodiment, after recording the videos and/orimages, the drones 125 can create an augmented view of the building byoverlaying sensor data onto the recorded video or images. At step 230,the drones transmit the received sensor data and the recordedimages/video to the monitoring computer 130. In embodiments in which thedrones 125 generate the augmented view, the augmented view istransmitted to the monitoring computer 130. In other embodiments, thedrones 125 may not receive any sensor data at the step 220. In suchembodiments, the step 220 is skipped and the drones capture the videosand/or images at the step 215 and transmit those videos/images back tothe monitoring computer 130 at the step 230. In such an embodiment, themonitoring computer can receive sensor information from, for example,the server 120 (or the gateway 115) and generate an augmented view usingthe images from the drones 125 and the sensor information from thesensors 135, 140, 145, 150, 155, and 160. The monitoring computer 130can receive the videos, images, and/or the augmented view from thedrones 125 and display the information for viewing by emergencypersonnel.

In an illustrative embodiment, the drones 125 continue to transmitinformation (e.g., the videos, images, and/or the augmented view) untilthe monitoring computer 130 directs the drones to stop receiving theinformation. The process 200 then ends at a step 235 with the dronesceasing to capture videos and/or images and traveling back to theemergency response crew.

Referring now to FIG. 3 , a flowchart outlining a process 300 forgenerating an augmented view on the monitoring computer 130 is shown, inaccordance with at least some embodiments of the present disclosure. Theprocess 300 is initiated at a step 305 with the building 105 catchingfire (or another undesirable condition), the emergency response crewbeing informed of the fire, and the emergency response crew traveling tothe location of the building. At the location of the building 105, theemergency response crew sends out the drones 125 in an operation 310. Asbriefly discussed above, the emergency response crew can program thedrones 125 before releasing them around the building 105.

For example, the emergency response crew programs the drones 125 with avariety of information using the monitoring computer 130. In oneexample, the drones 125 are programmed with the location of the building105. Upon arrival or en route to the building 105, the emergencyresponse crew programs the drones 125 with the location of the building105. The location may include the GPS coordinates of the building 105obtained from a sensor within the building that transmits location ofthe building, the gateway 115, the server 120, or from any other source(e.g., a location provided by a caller who called in an emergency to911). The location may also include the foot print (e.g., squarefootage) of the building 105, address, street intersections, anyspecific geographical or environmental conditions that may be relevant,physical structures or topology around the building, any peculiaritiesin the shape of the building, height of the building, and any otherinformation that the drones 125 may find useful in navigating, flying,and/or hovering around the building to capture videos and images in asafe and efficient manner. The emergency response crew may also get theinformation relating to the building 105 by connecting to the Internet.

The emergency response crew may also program the drones 125 by sendingsensor data from the sensors 135, 140, 145, 150, 155, and 160 to thedrones (if, for example, the drones do not have the capability toconnect with the sensors 135, 140, 145, 150, 155, and 160, the gateway115, and/or the server 120). The emergency response crew may alsoprogram the drones 125 with indications of whether to record videos,images, or both, as well as whether to record continuously orperiodically, whether to transmit the videos/images in real-time orperiodically, whether to hover around a particular area of the buildingor fly around the building, assigned area(s) of the building to capturevideos/images, how high or how further away from the building 105 tocaptures videos/images from, from what angles and directions to capture,etc.

Information specific to the building 105, such as, the height andfootprint of the building, floor-plan of the building, etc. may bepre-determined and pre-stored in the server 120 and, in someembodiments, the gateway 115. The server 120 and/or the gateway 115 mayalso include other information about the building 105 that themonitoring computer 130 may access, such as the age of the building, anyhazardous material in the building, any specific material used in theconstruction of the building, any specific fire safety features that thebuilding has, and any other feature that may aid the emergency responsecrew in devising the action plan. The gateway 115 and/or the server 120may also have stored thereon a layout or floor plan of the building 105,which the emergency response crew may download (e.g., access) and getacquainted with before devising the emergency action plan. The emergencyresponse crew may be able to simply connect to the server 120 (and/or tothe gateway 115 if the gateway is within a connection range) anddownload the location and building information for programming thedrones 125.

After programming the drones 125 and releasing the drones to capturevideos/images of the building 105, the drones may start capturingimagery of the building. This imagery, which can include videos and/orimages, is transmitted to the emergency response crew at a step 315. Asdiscussed above, the imagery transmitted by the drones 125 may includeraw videos/images that the drones 125 capture or, in some embodiments,an augmented view including the sensor data from the sensors 135, 140,145, 150, 155, and 160 in the videos/images. In addition to receivingthe imagery from the drones 125, the monitoring computer 130 receivesthe sensor data from the sensors 135, 140, 145, 150, 155, and 160 at astep 320. In some embodiments, if the monitoring computer 130 is unableto connect to the sensors 135, 140, 145, 150, 155, and 160, the gateway115, or the server 120, the monitoring computer may receive the sensordata from the drones 125. The monitoring computer 130 may also beconfigured to receive sensor data on demand, while in other embodiments,the monitoring computer 130 may continuously receive sensor data as newdata is reported.

In yet other embodiments, such as when the sensor data is onlyperiodically updated and transmitted to the monitoring computer 130, themonitoring computer may be configured to receive sensor data on demand.Thus, the monitoring computer 130 may be configured to receive sensordata both automatically and on demand. For on demand sensor data, theemergency response crew may choose to see a complete list of the sensorswithin the building 105 including a description of those sensors and thetype of information those sensors relay. Based on the list of sensors(and/or the imagery received from the drones 125), the emergencyresponse crew may identify a specific location or area of the buildingfrom which to receive more data from. If, in some instances, thelocation of the sensors 135, 140, 145, 150, 155, and 160 is nottransmitted, the emergency response crew may use GPS/Wi-Fi triangulationto communicate with the sensors to determine their location or get thelocation information from the drones 125.

Once the emergency response crew has the sensor data from the sensors135, 140, 145, 150, 155, and 160 and the imagery from all of the drones125, the monitoring computer 130 creates an augmented view at a step325. As discussed above, the drones 125 may have the capability tocreate an augmented view of the videos/images that the drone captures.If the monitoring computer 130 receives the augmented view from thedrones 125, the emergency response crew can use the augmented views fromall of the drones 125 to create an overall augmented view. In someembodiments, the creation of the overall augmented view happensautomatically, while in other embodiments, the emergency response crewcan decide which augmented views from the drones 125 to use to createthe overall augmented view. In an illustrative embodiment, the emergencyresponse crew has the ability to change the sensor information that isvisible on the overall augmented view. For example, the emergencyresponse crew can drag and drop sensors onto the imagery from the drones125 to see the information reported by those particular sensors.Likewise, the emergency response crew can remove sensor information fromthe overall augmented view if that sensor information is not deemedrelevant. The emergency response crew can view and change the augmentedreality view using the monitoring computer 130.

The overall augmented view can be displayed on a user interface of themonitoring computer 130 for review. The overall augmented view, in someembodiments, automatically provides suggestions for an action plan. Forexample, the augmented view may reveal that there are people or petstrapped inside the building 105 (e.g., via occupancy sensors). Theaugmented view may also reveal the best or closest and safest ways toextract the people/pets from the building 105 with minimal injuries tothe emergency response crew. The augmented view may also makesuggestions as to fire spreading patterns, which areas of the building105 are particularly vulnerable for explosions (e.g., by virtue ofhaving explosive devices or equipment in those areas), and so on. Basedon all the suggestions from the overall augmented view, the emergencyresponse crew and/or the monitoring computer 130 can devise an actionplan at step 335.

The emergency response crew and/or the monitoring computer 130 maycontinuously revise and refine the action plan based on the sensor datafrom the sensors 135, 140, 145, 150, 155, and 160 and the imagery fromthe drones 125. The process 300 then ends at step 340 with the emergencyresponse crew extinguishing the fire in the building 105 and/orcompleting any rescuing operations for rescuing occupants/property frominside the building.

Turning now to FIG. 4 , an illustrative monitoring computer 400 isshown, in accordance with at least some embodiments of the presentdisclosure. The monitoring computer 400 is similar to the monitoringcomputer 130 of FIG. 1 . The monitoring computer 400 includes, in someembodiments, a memory 405, a processor 410, a transceiver 415, a userinterface 420, and a power source 425. In alternative embodiments,additional, fewer, and/or different elements may be provided within themonitoring computer 400. The monitoring computer 400 can be a desktopcomputer, a laptop computer, a smartphone, another type of hand-helddevice, or a specialized computing device suitable for the functionsdescribed herein. The monitoring computer 400 can be used to implementone or more of the methods described herein.

In at least some embodiments, the memory 405 of the monitoring computer400 is an electronic holding place or storage for information so thatthe information can be accessed by the processor 410. The memory 405 mayinclude, but is not limited to, any type of random access memory (RAM),any type of read-only memory (ROM), dynamic random access memory (DRAM),programmable read only memory (PROM), erasable programmable read onlymemory (EPROM), electrically erasable programmable read only memory(EEPROM), any type of flash memory, such as magnetic storage devices(e.g., hard disk, floppy disk, magnetic strips, etc.), optical disks(e.g., compact disk (CD), digital versatile disk (DVD), etc.), smartcards, flash memory devices, etc. The monitoring computer 400 may alsohave one or more computer-readable media that use the same or adifferent memory media technology. The monitoring computer 4000 may haveone or more drives that support the loading of a memory medium such as aCD, a DVD, a flash memory card, etc.

Furthermore, the memory may be in communication with the processor 410that executes instructions. The instructions may be carried out by aspecial purpose computer, logic circuits, or hardware circuits. Theprocessor 410 may be implemented in hardware, firmware, software, or anycombination thereof. The term “execution” is, for example, the processof running an application or the carrying out of the operation calledfor by an instruction. The instructions may be written using one or moreprogramming language, scripting language, assembly language, etc. Theprocessor 410 executes an instruction, meaning that it performs theoperations called for by that instruction. The processor 410 operablycouples with the user interface 420, the transceiver 415, the memory405, etc. to receive, to send, and to process information and to controlthe operations of the monitoring computer 400. The processor 410 mayretrieve a set of instructions from a permanent memory device such as aROM device and copy the instructions in an executable form to atemporary memory device that is generally some form of RAM. Theprocessor 410 may include multiple processors that use the same or adifferent processing technology. In an illustrative embodiment, theinstructions may be stored in the memory 405.

With respect to the transceiver 415, the transceiver is configured toreceive and/or transmit information to/from the monitoring computer 400.For example, the transceiver 415 can be configured to communicate withthe drones 165, the server 155, the gateway 115, etc. In someembodiments, the transceiver 415 communicates information via a wiredconnection, such as an Ethernet connection, one or more twisted pairwires, coaxial cables, fiber optic cables, etc. In some embodiments, thetransceiver 415 communicates information via a wireless connection usingmicrowaves, infrared waves, radio waves, spread spectrum technologies,satellites, etc. The transceiver 415 can be configured to communicatewith another device using cellular networks, local area networks, widearea networks, the Internet, etc. In some embodiments, one or more ofthe elements of the monitoring computer 400 communicate via wired orwireless communications. In some embodiments, the transceiver 415provides an interface for presenting information from the monitoringcomputer 400 to external systems, users, or memory. For example, thetransceiver 415 may include an interface to a display, a printer, aspeaker, etc. In some embodiments, the transceiver 415 may also includealarm/indicator lights, a network interface, a disk drive, a computermemory device, etc. In other embodiments, the transceiver 415 canreceive information from external systems, users, memory, etc.

The user interface 420, in some embodiments, is configured to receiveand/or provide information from/to a user. The user(s) in this case arethe emergency response crew. The user interface 420 can be any suitableuser interface. The user interface 420 can be an interface for receivinguser input and/or machine instructions for entry into the monitoringcomputer 400. The user interface 420 may use various input technologiesincluding, but not limited to, a keyboard, a stylus and/or touch screen,a mouse, a track ball, a keypad, a microphone, voice recognition, motionrecognition, disk drives, remote controllers, input ports, one or morebuttons, dials, joysticks, etc. to allow an external source, such as auser, to enter information into the monitoring computer 400. The userinterface 420 can be used to navigate menus, adjust options, adjustsettings, adjust display, etc.

The user interface 420 can be configured to provide an interface forpresenting information from the monitoring computer 400 to externalsystems, users, memory, etc. For example, the user interface 420 caninclude an interface for a display, a printer, a speaker,alarm/indicator lights, a network interface, a disk drive, a computermemory device, etc. The user interface 420 can include a color display,a cathode-ray tube (CRT), a liquid crystal display (LCD), a plasmadisplay, an organic light-emitting diode (OLED) display, etc.

Furthermore, in some embodiments, the power source 425 is configured toprovide electrical power to one or more elements of the monitoringcomputer 130. In some embodiments, the power source 425 includes analternating power source, such as available line voltage (e.g., 120Volts alternating current at 60 Hertz in the United States). The powersource 425 can include one or more transformers, rectifiers, etc. toconvert electrical power into power useable by the one or more elementsof the monitoring computer 400, such as 1.5 Volts, 8 Volts, 12 Volts, 24Volts, etc. The power source 425 can include one or more batteries.

In an illustrative embodiment, any of the operations described hereincan be implemented at least in part as computer-readable instructionsstored on a computer-readable memory. Upon execution of thecomputer-readable instructions by a processor, the computer-readableinstructions can cause a node to perform the operations.

Furthermore, the monitoring computer 400 is configured to communicatewith sensors 430, gateway 435, server 440, and drones 445. The sensors430 are equivalent to the sensors 110 and specifically the sensors 135,140, 145, 150, 155, and 160. Likewise, the gateway 435 is similar to thegateway 115, the server 440 is similar to the server 120, and the drones445 are similar to the drones 125. The monitoring computer 400 isconfigured to receive sensor data from the sensors 430, the gateway 435,and the server 440, as well as captured videos and/or images from thedrones 445. The monitoring computer 4000 is configured to receiveinformation from and send information to the sensors 430, the gateway435, the server 440, and the drones 445 via communication links 450. Thecommunication links 450 are similar to the communication links 165 and170 of FIG. 1 . By virtue of communicating with the sensors 430, thegateway 435, the server 440, and the drones 445, the monitoring computer400 (e.g., the processor 410 of the device) can create an augmented viewfrom the sensor data of the sensors 430 and the videos/imagestransmitted by the drones 445. The augmented view can be viewed on theuser interface 420.

While the present disclosure has been described from the perspective ofa burning building, concepts of the present disclosure may be used inother cases as well. For example, features of this disclosure may beused for handling burglaries, to track burglars within a structure, tofacilitate rescue operations in other natural disasters, to monitor anactive shooter and the occupants inside, etc.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely exemplary, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected,” or“operably coupled,” to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable,” to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to inventions containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations. In addition, evenif a specific number of an introduced claim recitation is explicitlyrecited, those skilled in the art will recognize that such recitationshould typically be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). In those instances where aconvention analogous to “at least one of A, B, or C, etc.” is used, ingeneral such a construction is intended in the sense one having skill inthe art would understand the convention (e.g., “a system having at leastone of A, B, or C” would include but not be limited to systems that haveA alone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). It will be furtherunderstood by those within the art that virtually any disjunctive wordand/or phrase presenting two or more alternative terms, whether in thedescription, claims, or drawings, should be understood to contemplatethe possibilities of including one of the terms, either of the terms, orboth terms. For example, the phrase “A or B” will be understood toinclude the possibilities of “A” or “B” or “A and B.” Further, unlessotherwise noted, the use of the words “approximate,” “about,” “around,”“substantially,” etc., mean plus or minus ten percent.

The foregoing description of illustrative embodiments has been presentedfor purposes of illustration and of description. It is not intended tobe exhaustive or limiting with respect to the precise form disclosed,and modifications and variations are possible in light of the aboveteachings or may be acquired from practice of the disclosed embodiments.It is intended that the scope of the invention be defined by the claimsappended hereto and their equivalents.

1-2. (canceled)
 3. A method of monitoring a structure with an augmentedreality system, the method comprising: receiving, at a self-propelledmonitoring device, instructions to monitor an assigned area associatedwith a monitored structure; in response to the instructions to monitorthe assigned area, the self-propelled monitoring device traveling to theassigned area; receiving, by the self-propelled monitoring device,sensor data from one or more building sensors associated with theassigned area; determining one or more vantage points from which tomonitor the assigned area, the determination based on the receivedsensor data; capturing, by the self-propelled monitoring device, imagedata of the assigned area from at least one of the vantage points; andcreating, by the self-propelled monitoring device, an augmented view ofthe monitored structure, wherein the augmented view incorporates,occupancy data, the image data of the assigned area, a building layout,and the sensor data from the one or more building sensors.
 4. The methodof claim 3, wherein the self-propelled monitoring device comprises anaerial drone.
 5. The method of claim 3, wherein determining the one ormore vantage points is based on a safety of the vantage point, anavailability of sensor data from the vantage point, and an availabilityof image data from the vantage point; and wherein the image data of themonitored structure comprises at least one of real-time video and one ormore still images.
 6. The method of claim 3, wherein creating theaugmented view of the monitored structure comprises selectivelyoverlaying at least a portion of the sensor data on the image data ofthe monitored structure.
 7. The method of claim 3, wherein the sensordata comprises broadcast information indicative of a sensed condition,an identity code associated with each of the respective buildingsensors, and location information for each of the respective buildingsensors.
 8. The method of claim 3, further comprising transmitting theaugmented view to a monitoring computer for display.
 9. An augmentedreality system, the system comprising: a monitoring computer configuredto receive information regarding a monitored structure, the informationcomprising a building layout; and a self-propelled monitoring devicecommunicatively coupled to the monitoring computer, wherein the movablemonitoring device is configured to: receive, from the monitoringcomputer, instructions to monitor an assigned area associated with themonitored structure; in response to the instructions, travel to theassigned area; receive sensor data from one or more of the buildingsensors associated with the monitored structure; determine one or morevantage points from which to monitor the assigned area based on receivedsensor date from at least one of the building sensors; travel to atleast one of the vantage points; and capture image data of the assignedarea associated with the monitored structure; wherein the self-propelledmonitoring device is further configured to create an augmented view ofthe monitored structure, wherein the augmented view selectivelyincorporates the building layout, the image data of the assigned area,and the sensor data from the one or more building sensors, the augmentedview further incorporating an indication of whether the monitoredstructure is occupied.
 10. The system of claim 9, wherein the monitoringcomputer is communicatively coupled to a server, and wherein themonitoring computer is configured to receive layout information for themonitored structure from the server.
 11. The system of claim 10, whereinone of the monitoring computer and the self-propelled monitoring deviceis configured to determine the instructions to travel to the vantagepoint based on the layout information received from the server.
 12. Thesystem of claim 9, further comprising the one or more building sensors,wherein the one or more building sensors are configured to broadcast thesensor data as a signal at regular intervals, and wherein the signalcomprises identification information for each of the one or morebuilding sensors and data indicative a sensed condition.
 13. The systemof claim 9, wherein the self-propelled monitoring device comprises anaerial drone.
 14. The system of claim 9, wherein the sensed environmentassociated with the structure comprises at least one of a safety of thevantage point, an availability of the sensor data from the vantagepoint, and an availability of the image data from the vantage point; andwherein the image data of the monitored structure comprises real-timevideo.
 15. The system of claim 9, wherein the self-propelled monitoringdevice is further configured to at least one of (a) determine the sensedenvironment based on the received sensor data from the one or morebuilding sensors; or (b) monitor at least one of the structure andsurroundings of the structure to determine the sensed environment. 16.The system of claim 15, wherein the self-propelled monitoring device isfurther configured to: receive building data representing buildingshape, height, and structures or topology around the building; anddetermine the one or more vantage points based on the received sensorand building data.
 17. A method of monitoring a structure with anaugmented reality system, the method comprising: transmitting, by amonitoring computer to a self-propelled monitoring device, instructionsto monitor an assigned area associated with a monitored structure;traveling, by the self-propelled monitoring device, to the assignedarea; receiving, by the self-propelled monitoring device, sensor datafrom one or more building sensors associated with the assigned area;receiving, by the monitoring computer from the self-propelled monitoringdevice, the sensor data from one or more building sensors associatedwith the assigned area; determining one or more a vantage points fromwhich to monitor the assigned area based on the sensor data from one ormore building sensors associated with the assigned area; receiving, bythe monitoring computer from the self-propelled monitoring device, imagedata of the assigned area; creating, by the monitoring computer, anaugmented view of the monitored structure, wherein the augmented viewselectively incorporates the image data of the assigned area, afloorplan of the structure, and the sensor data from the one or more ofthe building sensors; and determining the severity of a hazardouscondition using sensor data and image data and detecting a rate ofspread of the hazardous condition.
 18. The method of claim 17, whereinthe sensor data comprises broadcast information indicative of a sensedcondition, an identity code associated with each of the respectivebuilding sensors, and location information for each of the respectivebuilding sensors.
 19. The method of claim 17, further comprising:receiving, by the monitoring computer, information regarding a layout ofthe monitored structure; and determining the instructions to travel tothe destination based on the information regarding the layout of themonitoring structure.
 20. The method of claim 17, further comprising:displaying, by the monitoring computer, a list of the one or morebuilding sensors; receiving, at the monitoring computer, a selection ofa specific location within the monitored structure from which to receiveadditional data; and presenting, by the monitoring computer, additionalinformation regarding the specific location in response to the selectionof the specific location.
 21. The method of claim 17, further comprisingdetermining and presenting, by the monitoring computer, a proposedaction plan concurrently with the augmented view.
 22. The method ofclaim 17, wherein the self-propelled monitoring device comprises atleast one of an aerial drone and a terrestrial drone.
 23. The method ofclaim 17, wherein the augmented view comprises image data of themonitored structure having the sensor data overlaid thereon.